Parametric models — definitions
Quantifying parameter uncertainty using probability distributions
Questions we can answer better by quantifying uncertainty
Uncertainty analysis — how confident are our results, how strong is the evidence?
Sensitivity analysis — how would results change if part of the model changed?
Value of Information (VoI) analysis — what parts of the model do we need better evidence for?
Models approximate process that generates some output of interest, helping to inform decision-making.
Example: ITHIM health impact model [ TODO borrow picture ]
inputs: travel data, physical activity, air pollution, traffic injuries
scenarios of change in these
outputs: population health (and changes under scenarios)
These generally involve parameters. What are these?
Model input parameters: examples
background exposure, e.g. average levels of an air pollutant, or levels of physical activity (for some population, area, characteristics)
relative risk of getting a disease, given some change in some exposure
Parameters are representations of knowledge about a (real or imagined) population.
Model transform parameters \(\rightarrow\) output quantities.
Outputs also represent summaries over populations
What is a parameter and why are parameters uncertain (5 min)
Two broad reasons for parameter uncertainty: the parameter estimate is
an observed summary of a limited population, and/or
from a population that is different from the one we want
Models are also uncertain (“structural” uncertainty)
By definition, model approximates reality (“all models are wrong, but some are useful”…)
Ideally quantify as many uncertainties as possible inside the model
| Individual-level quantities | Parameters |
|---|---|
| individual exposure to PM2.5 during a trip | expected exposure to PM2.5 for a trip/individual of this kind |
| whether this individual dies within a year | risk of death within a year |
Individual-level quantities are different for each individual
Our knowledge of parameters may be uncertain
Individual variability can never be removed, but parameter uncertainty can be reduced with better knowledge
We’ve built best model we can. Just report our “best estimate” to the decision maker?
Decision makers need good evidence to change practice — models should indicate strength of evidence for result
What about the future - we may be able to get better evidence - what research should be done?
Here we will cover quantitative methods for assessing uncertainty. In particular, probabilistic methods.
Uncertainty analysis: about strength of evidence
What range of outputs are plausible, given current evidence?
Which parameters most influence the output uncertainty?
Sensitivity analysis: “what if…”
Value of Information analysis
(advanced, may remove this slide)
Microsimulation models create a population of synthetic individuals and simulating / summarising quantities of interest
Often based on drawing individuals directly from an observed dataset
ITHIM: synthetic population of “individuals” (households, people, trips) based on travel survey data. Exposures assigned to trips. Health impacts for aggregated age groups.
Uncertainty here applies to individual-level quantities (e.g. trips taken for a person with given characteristics), and is represented by variability between individuals in the survey data. This approach doesn’t consider uncertainty about quality/relevance of the travel survey dataset
Two broad approaches
Statistical analyses of data (your own, or published analyses) giving point and interval estimates or standard errors
Judgements (informal or from structured expert elicitation), e.g.
point estimate (best guess) for parameter value
credible interval e.g. “I judge that the parameter is between \(a\) and \(b\), with 95% confidence”
Our goal in each case is to obtain probability distributions for parameters
A full probability distribution
For any pair of values \((a,b)\), we can deduce the probability that the parameter is between \(a\) and \(b\).
In statistical models, data assumed to come from models with parameters…
e.g. number of disease cases is Binomial (population n, underlying prevalence p)
Bayesian methods based around quantifying parameter uncertainties as probability distributions
Prior distribution combined with study data \(\rightarrow\) posterior distribution.
Prior distribution dominates if data are weak
Prior doesn’t matter if enough data
Construct an estimator of parameters based on a finite dataset
Uncertainty quantified by imagining different datasets drawn from the same population
Standard error: variability in estimates between datasets.
Confidence interval: contains true value in 95% of datasets drawn
If dataset is large, can interpret a frequentist analysis as Bayesian: parameter has a normal distribution, defined by estimate and standard error
If dataset is small, Bayesian analyses have the benefit of allowing background information to be included as a prior
Global
Epidemiological models for estimating disease incidence and mortality
Meta-analysis of relative risks (of disease outcomes given different exposures)
Local
Road traffic deaths or injuries: count data (e.g. Poisson) regression models of reported incidents by person/vehicle characteristics
Models for analysing travel survey data
MMET value for cycle commuting. Data from Compendium of Physical Activities (https://pacompendium.com/bicycling/)
Which of these are relevant to our particular health impact model?
Might judge that code 1011 (MET 6.8) is a typical kind of cycling for our context, and judge a credible interval based on similar kinds of cycling in this table.
Rough judgement: no “correct answer”!
Note: Does the parameter in our model represent the average for some population, or the value for a person / trip?
Note also: Each published MMET value is itself an estimate, that conceals variability and uncertainty!
PM2.5 concentration. We know published estimates of average and SD for 30 cities in India.
Suppose our city is not one of these. We just know the average PM2.5 for our city. Where do we get a credible interval or SD?
Indian data shows a “typical” set of SDs of pollution within cities. So we might use an average of these SDs. But…
What do we know about how our city compares to these?
Do we want a measure of uncertainty (about an average) or variability (between points)?
Source: https://urbanemissions.info/wp-content/uploads/apna/docs/2019-07-APnA30city_summary_report.pdf
Formally averaging a quantity estimated from different studies.
Give more weight to more confident studies and/or those closer to our context.
Most developed in randomised clinical trials — highly controlled, regulated studies
see e.g. https://training.cochrane.org/interactivelearning for learning resources — find those about observational studies in epidemiology
Might summarise the average of the observed studies, or make a prediction for a new study (more uncertain)
If studies disagree, consider why. Ensure relevant studies selected.
Not always possible to precisely quantify uncertainty
Rough, clearly-stated judgement better than ignoring it
Uncertainty about a parameter may or may not affect the uncertainty about a model output (see “Value of Information”, later…)
When making judgements, consider, e.g.:
quality/amount of data behind published numbers. how were these numbers obtained?
relevance of population, variations over time / between places
Quantifying judgements with probability (5 min)
Suppose we have a parameter with estimate 0, credible interval (-2 to 2)
How to derive a probability distribution that reflects this belief?
What is wrong with a distribution like this
Suppose we have a parameter with estimate 0, credible interval (-2 to 2)
A triangular distribution is a bit more plausible
Suppose we have a parameter with estimate 0, credible interval (-2 to 2)
A normal distribution is even better
Used for quantities with unrestricted ranges
Defined by mean \(\mu\) and standard deviation \(\sigma\) (or variance \(\sigma^2\))
95% credible interval is \(\pm 2\) SDs: i.e. width is 4 SDs.: SD easily derived from a CI.
Used for positive-valued quantities.
Normal distribution for the log of the quantity
Example: MMET/h estimate 2.5 (CI 1 to 4).
Transform to log(MMET).
Estimate log(2.5), CI (log(1) to log(4)).
Assign normal with SD = CI width / 4
If the SD is published, but not the CI
e.g. MMET/h estimate \(m=2.5\) (SD \(s=1.4\))?
Method of moments gets us mean \(\mu\) and SD \(\sigma\) on log scale
\(\mu = \log(m/\sqrt{s^2/m^2 + 1}), \sigma= \sqrt{\log(s^2/m^2 + 1)}\)
Used for quantities between 0 and 1: probabilities, proportions
Defined by “shape” parameters \(a,b\)
Given an estimate \(m\) and credible interval, how to obtain \(a, b\)?
Approximate SD \(s =\) CI width/4, then use “method of moments”:
\(a = (m(1-m)/s^2 - 1)m\)
\(b = (m(1-m)/s^2 - 1)(1 - m)\)
“SD = CI width / 4” heuristic based on symmetric, normal-like distributions.
Less accurate in example (b) opposite.
Could adjust by trial and error to match desired belief more closely.
See SHELF R package for more sophisticated techniques for fitting distributions
Obtaining full probability distributions from published estimates and uncertainties (20 min)
For each \(i = 1, 2, \ldots N\) (enough to give precise summaries)
Simulate parameters \(X_i\) from their uncertainty distributions
Compute the model output \(Y_i = g(X_i)\)
producing a sample from the model outputs \(Y_1, \ldots, Y_N\) [ animate? ]
Summarise the sample to give e.g.
a credible interval for the outputs
probability that e.g. number of deaths \(>\) [important value]
As \(N\) gets greater, the summaries of the sample will not change much when you run more.
run enough until you have the precision you want
do you really need more than 2 or 3 significant figures?
More precisely: the Monte Carlo standard error (MCSE) describes how accurate a summary of a Monte Carlo sample is.
For the sample mean, the MCSE is \(SD(sample) / \sqrt{N}\)
This reduces to zero as \(N\) gets larger.
We can then say the mean of the sample is precise to \(\pm\) 2 MCSE
If your model is very slow and doing Monte Carlo is too expensive to get the precision you want, then present MCSE alongside the estimate to show your readers how much precision you do have.
Given a model with uncertain inputs \(X\), with expected values \(E(X)\)
Outputs \(Y = g(X)\) where \(g()\) is function that defines the model
What is \(E(Y)\), the expected value of \(Y\)?
Can we just plug in the best estimates of the inputs, as \(g(E(X))\)?
\(E(Y)\) only equals \(g(E(X))\) if the function \(g()\) is linear:
Proof
\(g(E(x)) = g((X_1 + \ldots + X_n)/n)\), then if \(g()\) is linear, this equals
\((g(X_1) + \ldots + g(X_n)) / n = E(g(X)) = E(Y)\).
Most realistic models are non-linear, therefore need Monte Carlo simulation to get the true expectation of the output.
Answers questions like “what if [parameter] was actually \(b\) instead of \(a\)”
For each parameter, compare model output with
parameter at a “low value”
parameter at a “high value”
What if all the other parameters are uncertain? Probabilistic one-way sensitivity analysis:
Fix [parameter] at high or low value
Run the model under Monte Carlo analysis, using the uncertainty distributions of the other parameters
Tornado plot
Arbitrary choice of “low” and “high” value may not convey the effect of the parameter
e.g. if effect of parameter is nonlinear
What if parameters have correlated effect on the outcome?
Hard to vary more than one parameter at a time
Monte Carlo simulation of health impact models (30 min)
Recall
sensitivity analysis: “what if model was a bit different”
uncertainty analysis: “what is strength of evidence in model”
A different (related) question: “what would be the benefit of getting better information”.
This is Value of Information analysis
Given current information, model output is uncertain. But how much more precise would it get…
if we were to learn some parameter exactly?
if we conducted a study (say, a survey of 100 people) to estimate it
Helps us to:
set research priorities to reduce uncertainty
design studies (more advanced, not covered here)
Value of Information methods developed in health economics
Model a health policy decision and its consequences (e.g. health benefits as QALYs vs costs). Parameter uncertainties as probability distributions
Information has value: \(\rightarrow\) reduces parameter uncertainty \(\rightarrow\) more precise model outputs \(\rightarrow\) better informed policy-making \(\rightarrow\) health benefits
Not previously used much in health impact modelling, where models used more for scenarios than policies
How can we define “value” if we don’t model a health policy?
Define value as “precision of estimate” (e.g. health impact of scenario)
How much will the variance (or SD, or credible interval width) be expected to reduce if we got better information?
More precise estimates (implicitly) assumed to ultimately lead to benefits
If needed, trade off informally with costs of research (willingness to pay for more precise estimates? Not covered here)
How exactly to do these computations?
Model \(Y = g(X_1, X_2,.. )\) with some (scalar) output \(Y\) and multiple inputs \(X_r\)
Do uncertainty analysis: define distributions for each \(X_r\), obtain distribution for \(Y\) via Monte Carlo simulation.
\(var(Y)\): variance of model output under current information
Definition of EVPPI - expected reduction in this variance if we were to learn the exact value of \(X_r\) (say)
\[var(Y) - E_x(var(Y | X_r = x))\]
We don’t know the value of \(X_r\) when we calculate this — so we must take the expectation over possible values \(x\) (using the distribution we defined)
Take the Monte Carlo sample, with all parameters uncertain
Regression of \(Y\) = model output versus \(X\) = parameter of interest
Expected reduction in variance of \(Y\) if we learnt \(X\)
\[var(Y) - E_x(var(Y | X = x))\]
We don’t know \(X\) so we average over our uncertainty
\(var(Y)\): uncertainty before learning \(X\)
\(E_x(var(Y | X = x))\): residual variance: mean of (“observed” - fitted) over different \(x\)
Regression function of output on input will not necessarily be linear.
Spline regression (automated choice of smoothness, works well enough in this context)
lm(Y ~ X) # linear model
library(mgcv)
gam(Y ~ X) # spline ("generalized additive") model
The voi package can take care of extracting ingredients needed for the EVPPI computation (see the practical)
library(voi)
evppivar(outputs, inputs)
Example: dose-response curve governed by four different parameters \(\alpha\), \(\beta\), \(\gamma\), \(\tau\). Estimate the expected value of jointly learning all four of them (hence learning the dose-response curve)
Regression model with four predictors, e.g.
gam(Y ~ alpha + beta + gamma + tau)
More advanced regression models available, which automatically choose the best fitting regression function for Y given the predictors (and their interactions): see the practical about the voi package
Impact of different parameters on the output uncertainty, presented as
Uncertainty that would remain in the output \(Y\) if we knew \(X\)
as a variance, \(var(Y) - EVPPI_X\)
…or standard deviation (square root of this)
…or rough credible interval (\(\pm 2\) remaining standard deviation)
or proportion of uncertainty (variance) in \(Y\) explained by \(X\)
We will never get perfect information, but we may get better information via a new research study.
Expected value of sample information is the expected value of a study of a particular design / sample size
Slightly harder to define and compute
Previously used in health economic decision models, but not (yet!) used in health impact models
See voi package and references there for some examples.
Here we have just talked about “value” as variance of outputs
Value of Information also used widely for health economic decision models
Model chooses policy with the highest net benefit (or lowest loss)
“Value” is the net benefit (NB) itself
Value of information is NB(better information) - NB(current information)
See voi package and references there for some examples.
Value of Information analysis, using the voi package
(Worked example, 20 min)
What uncertainty/sensitivity questions do you want to answer in your own work?
Do you have the tools and skills to do this, after this course?